Work estimation apparatus, method and non-transitory computer-readable storage medium

ABSTRACT

According to one embodiment, a work estimation apparatus includes processing circuitry. The processing circuitry acquires video data on a predetermined area, calculates a work value related to work performed by a worker included in the video data, based on the video data, and displays the work value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2020-154120, filed Sep. 14, 2020, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a work estimationapparatus, a method and a non-transitory computer-readable storagemedium.

BACKGROUND

Conventionally, there is known a technique for acquiring the physicalactivity state of a worker by use of a wearable device, such as awearable sensor and a wearable camera.

The wearable device, however, needs to be attached to the worker, sothat it may be a hindrance to the work performed by the worker.Therefore, acquiring a work load of the worker by the wearable devicemay be a burden on the part of the worker who performs the work. Inaddition, there are concerns about the problem of the cost required forintroducing the same number of wearable devices as the number ofworkers, and the problem of the maintenance and charging of the wearabledevices.

There is also known a technique of associating the position of a workerin a work place with a work load of the worker acquired by a wearabledevice and visualizing them as a heat map on a sketch.

However, the heat map associates the position of the worker with theactivity state of the worker, so that the work load on the equipmentplaced in the work place cannot be visualized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration example of a workestimation system including a work estimation apparatus according to afirst embodiment.

FIG. 2 is a block diagram showing a configuration example of aprocessing unit of the work estimation apparatus according to the firstembodiment.

FIG. 3 is a flowchart illustrating an operation example of the workestimation apparatus according to the first embodiment.

FIG. 4 is a flowchart illustrating a specific example of how the workvalue calculation processing is performed according to the firstembodiment.

FIG. 5 is a diagram illustrating a two-dimensional human skeleton modelaccording to the first embodiment.

FIG. 6 is a diagram illustrating how a two-dimensional human skeletonmodel and a three-dimensional human skeleton model are estimated from animage according to the first embodiment.

FIG. 7 is a table illustrating how states of a plurality of body partsare classified according to the first embodiment.

FIG. 8 is a first example of how the posture of a worker is estimatedfrom an image according to the first embodiment.

FIG. 9 is a second example of how the posture of a worker is estimatedfrom an image according to the first embodiment.

FIG. 10 is a third example of how the posture of a worker is estimatedfrom an image according to the first embodiment.

FIG. 11 is a fourth example of how the posture of a worker is estimatedfrom an image according to the first embodiment.

FIG. 12 is a table in which a combination of states of a plurality ofbody parts and load values are associated with each other according tothe first embodiment.

FIG. 13 is a diagram illustrating how types of load values are in thefirst embodiment.

FIG. 14 is a human body diagram illustrating how accumulated load valuesare displayed in correspondence to a plurality of body parts accordingto the first embodiment.

FIG. 15 illustrates a plurality of human body diagrams illustrating howaccumulated load values are displayed in correspondence to a pluralityof body parts according to the first embodiment, the human body diagramsbeing arranged in time series.

FIG. 16 is a block diagram showing a configuration example of a workestimation system including a work estimation apparatus according to asecond embodiment.

FIG. 17 is a block diagram showing a configuration of a processing unitof the work estimation apparatus according to the second embodiment.

FIG. 18 is a flowchart showing an operation example of the workestimation apparatus according to the second embodiment.

FIG. 19 is a flowchart showing a specific example of how the work targetestimation processing is performed according to the second embodiment.

FIG. 20 is a diagram illustrating an image of video data according tothe second embodiment.

FIG. 21 is a diagram illustrating a two-dimensional sketch according tothe second embodiment.

FIG. 22 is a diagram illustrating three reference points in the imageshown in FIG. 20.

FIG. 23 is a diagram illustrating three reference points in the sketchshown in FIG. 21.

FIG. 24 is a diagram for illustrating how the image shown in FIG. 20 andthe sketch shown in FIG. 21 are related to virtual three-dimensionalspace.

FIG. 25 is a diagram illustrating how the coordinates of referencepoints in the image and the coordinates of reference points in thesketch are converted to coordinates of reference points of the virtualthree-dimensional space according to the second embodiment.

FIG. 26 is a diagram for illustrating how a three-dimensional humanskeleton model represented by normalized three-dimensional coordinatesis arranged in the virtual three-dimensional space according to thesecond embodiment.

FIG. 27 is a diagram illustrating how the coordinates of key points ofthe three-dimensional human skeleton model represented by the normalizedthree-dimensional coordinates are converted to the coordinates of thekey points of the three-dimensional human skeleton model represented inthe virtual three-dimensional space according to the second embodiment.

FIG. 28 is a diagram for illustrating how the direction of the worker isestimated according to the second embodiment.

FIG. 29 is a diagram for illustrating how the direction of the worker isassociated with work targets in the sketch according to the secondembodiment.

FIG. 30 is a sketch illustrating how accumulated load values aredisplayed in correspondence to a plurality of work targets according tothe second embodiment.

FIG. 31 is a diagram illustrating a first specific example of displaydata obtained according to the second embodiment and including a humanbody diagram showing a specific body part and a sketch showing loadvalues related to the specific body part and displayed in correspondenceto a plurality of work targets.

FIG. 32 is a diagram illustrating a second specific example of displaydata obtained according to the second embodiment and including a humanbody diagram showing a specific body part and a sketch showing loadvalues related to the specific body dart and displayed in correspondenceto a plurality of work targets.

FIG. 33 is a diagram illustrating a third specific example of displaydata obtained according to the second embodiment and including a humanbody diagram showing a specific body part and a sketch showing loadvalues related to the specific body part and displayed in correspondenceto a plurality of work targets.

FIG. 34 is a diagram illustrating display data including a sketchshowing a specific work target and a human body diagram in whichaccumulated load values are displayed in correspondence to a pluralityof body parts.

FIG. 35 is a block diagram illustrating a hardware configuration of acomputer according to one embodiment.

DETAILED DESCRIPTION

In general, according to one embodiment, a work estimation apparatusincludes processing circuitry. The processing circuitry acquires videodata on a predetermined area, calculates a work value related to workperformed by a worker included in the video data, based on the videodata, and displays the work value.

Embodiments of the work estimation apparatus will now be described indetail with reference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing a configuration example of a workestimation system 1 including a work estimation apparatus 100 accordingto the first. embodiment. The work estimation system 1 includes a workestimation apparatus 100, a photographing device 200, a storage device300 and an output device 400. The work estimation system 1 is used topresent (visualize) a state related to the work of a worker in a targetarea in association with a place and an object in the target area, in anexpression format that is easy for the user to recognize.

The photographing device 200 is, for example, a video camera. Thephotographing device 200 photographs a work area (e.g. an assembly workplace of a factory) where work is being performed by a worker, andacquires a still image or a moving image. In the present embodiment, thestill image or moving image acquired by the photographing device 200 isreferred to as video data. The photographing device 200 outputs theacquired video data to the storage device 300. The photographing device200 may output the acquired video data directly to the work estimationapparatus 100. The video data may include a photographing time.

The storage device 300 is a computer-readable storage medium that storesdata in a nonvolatile manner. This storage medium is, for example, aflash memory, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).The storage device 300 stores video data output from the photographingdevice 200. Further, the storage device 300 stores, for example, aplurality of data used in the work estimation apparatus 100. Theplurality of data include, for example, posture estimation data, workvalue calculation data, history data, display conversion data, etc.Details of each of the plurality of data will be described later. Thestorage device 300 outputs video data and a plurality of data to thework estimation apparatus 100 in response to access from the workestimation apparatus 100.

The video data stored in the storage device 300 may be associated, forexample, with calendar information or Gantt chart information. Thestorage device 300 may be provided in an external server. The storagedevice 300 may include a plurality of storage media. Each of theplurality of storage media may be used properly according to the type ofdata which is to be stored. Specifically, the HDD or SSD used as astorage medium may store video data that is large in data amount, andthe flash memory used as a storage medium may store the above-mentionedplurality of data that are small in data amount.

The work estimation apparatus 100 is, for example, a computer used by asystem administrator (hereinafter, referred to as a user) who managesthe work estimation system 1. The work estimation apparatus 100 includesan acquisition unit 110, a processing unit 120 and a display controlunit 130. The work estimation apparatus 100 may include at least one ofthe photographing device 200, the storage device 300 and the outputdevice 400.

The acquisition unit 110 acquires video data from the storage device300. The acquired video data may be video data that are photographed inreal time by the photographing device 200 and sequentially stored in thestorage device 300, or may be video data that are stored in advance inthe storage device 300. The acquisition unit 110 outputs the acquiredvideo data to the processing unit 120. The acquisition unit 110 mayacquire video data directly from the photographing device 200. Forexample, the acquisition unit 110 may collectively acquire moving imagesas one data, or may sequentially acquire moving images in a streamingformat.

The processing unit 120 receives video data from the acquisition unit110. The processing unit 120 calculates a work value related to theposture of a worker (working posture), based on the video data. The workvalue may be paraphrased as a value related to the work performed by theworker. By accessing the storage device 300, the processing unit 120 mayreceive a plurality of data necessary for processing the video data. Theprocessing unit 120 may cause the storage device 300 to store thecalculated work value as it is, or may cause the storage device 300 tostore the calculated work value in association with the video data orinformation related to the video data (video data information). Thevideo data information includes, for example, a file name of the videodata and frame information related to frames constituting the videodata. By causing the storage device 300 to store the work value inassociation with the video data or the video data information, the workestimation apparatus 100 can perform useful visualization andcalculation of useful statistical values.

The work value includes, for example, at least a value (load value)representing the physical load of the worker. When the processing unit120 calculates the load value, the processing unit 120 calculates, forexample, a posture feature amount of the worker, based on the videodata, estimates a working posture of the worker, based on the calculatedposture feature amount, and calculates a load value, based on theworking posture. The work value may include a value indicative of thetype of work and a value indicative of the attendance state of theworker. The work estimation apparatus 100 may present these values tothe user in association with the load values. This can be useful forbusiness improvement.

The processing unit 120 generates display data in an expression formatthat is easy for the user to recognize, based on the calculated workvalue. The display data according to the first embodiment is, forexample, statistical data (described later) of work values calculatedfor a plurality of body parts (e.g., the back, the upper limbs, thelower limbs, etc.) of a human body diagram that is regarded as a worker.Specifically, the display data according to the first embodiment isobtained by superimposing a map corresponding to the statistical datadescribed later on the plurality of body parts of the human bodydiagram. The processing unit 120 outputs the generated display data tothe display control unit 130. The processing unit 120 may cause thestorage device 300 to store the generated display data as it is, or maycause the storage device 300 to store the generated display data inassociation with the video data or the video data information. Thedisplay data may include numerical values included in the statisticaldata, figures corresponding to the numerical values and a table or agraph that is based on the statistical data, such that they are shown onthe human body diagram or in the neighborhood thereof.

The display control unit 130 receives display data from the processingunit 120. The display control unit 130 causes the output device 400 todisplay the display data.

FIG. 2 is a block diagram showing a configuration example of theprocessing unit 120 of the work estimation apparatus 100 according tothe first embodiment. The processing unit 120 includes a postureestimation unit 121, a work value calculation unit 122, a statisticalprocessing unit 123 and a display data generation unit 124.

The posture estimation unit 121 estimates a posture of the worker, basedon the video data. Specifically, the posture estimation unit 121 detectsa worker from the video data, using posture estimation data, andestimates a posture of the detected worker. The posture estimation unit121 outputs information on the estimated posture of the worker to thework value calculation unit 122.

The posture estimation data include, for example, a learned machinelearning model (hereinafter, referred to simply as a “learned model”)trained to detect a person from video data, a learned model trained toestimate a posture of a person, and a learned model trained to performboth. As these learned models, a Neural Network (NN) is used, andpreferably a Convolutional Neural Network (CNN) is used. By using theCNN, a target (a person in the present embodiment) can be accuratelydetected from an image. The learned model is not limited to the CNN, andvarious other NNs suitable for intended use (e.g., Graph Neural Network(GNN) 3D-CNN, etc.) may be used. This holds true of the processingdescribed below.

The learned models used to estimate the posture of a person include, forexample, a two-dimensional skeleton estimation model that estimates askeleton of a person of video data on a two-dimensional image, athree-dimensional skeleton estimation model that estimates athree-dimensional skeleton by applying a two-dimensional skeletonestimation result (corresponding to a “two-dimensional human skeletonmodel” described later) to normalized three-dimensional coordinates, anda behavior estimation model that estimates a behavior of a person fromtime-series data on a three-dimensional skeleton estimation result(“three-dimensional human skeleton model” described later). Thetwo-dimensional skeleton estimation model and the three-dimensionalskeleton estimation model may be collectively referred to as a skeletonestimation model. Both the skeleton estimation result and the humanskeleton model may be paraphrased as posture features.

The two-dimensional skeleton estimation model is trained in advance suchthat a person can be detected from video data and a skeleton can bedetected from the detected person. The three-dimensional skeletonestimation model is trained in advance such that a three-dimensionalskeleton can be estimated from the skeleton of a person on atwo-dimensional image. The behavior estimation model is trained inadvance such that a behavior and a posture of a person can be estimatedfrom the time series data on the three-dimensional skeleton. Thebehavior estimation model is not limited to this. For example, a modelthat determines whether or not a person in a still image is walking witha reliability from zero to 1 may be used. Further, a three-dimensionalskeleton estimation model that estimates a three-dimensional humanskeleton model of a person directly from video data may be used withoutusing a two-dimensional skeleton estimation model. In this case, atwo-dimensional human skeleton model may be estimated from thethree-dimensional human skeleton model.

The posture estimation unit 121 may specify a worker, based on the videodata. Specifically, the posture estimation unit 121 identifies adetected worker by using worker identification data. The workeridentification data includes for example, a learned model (workeridentification model) trained to identify a worker from video data. Theworker identification model is trained in advance such that the workercan be identified from a face photograph of a worker and a photograph ofthe clothes of the worker.

The work value calculation unit 122 receives information on the postureof a worker from the posture estimation unit 121. The work valuecalculation unit 122 calculates a work value of the worker, based on theposture of the worker. Specifically, the work value calculation unit 122calculates the work value of the worker corresponding to the posture ofthe worker, by using work value calculation data. The work valuecalculation unit 122 outputs the calculated work value of the worker tothe statistical processing unit 123.

The work value calculation data includes, for example, a learned modeltrained to calculate a work value from the posture of a person, a tablein which the posture of the person and the work value are associatedwith each other, etc. The work value calculation data may include a facephotograph of each of a plurality of workers, a work process chart ofeach of the plurality of workers, a table in which a work and a workplace are associated with each other, a photograph of the clothes ofeach of the workers, etc. Thus, the work value calculation unit 122 mayperform processing for identifying a plurality of workers, processingfor specifying the work of each worker and specifying a work place. Itshould be noted that these processes may be performed by a unit otherthan the work value calculation unit 122.

The statistical processing unit 123 receives a work value of a workerfrom the work value calculation unit 122. The statistical processingunit 123 generates statistical data regarding the work value of theworker, based on the work value. Specifically, the statisticalprocessing unit 123 generates, as statistical data, work valuesaccumulated from the start of work to each arbitrary point of time,based on the work period of the worker and the work value correspondingto each point of time of the work period. Where the work value is a loadvalue, the work value (load value) may be generated for each of aplurality of body parts. The statistical processing unit 123 outputs thegenerated statistical data to the display data generation unit 124.

When an accumulated work value is calculated, the statistical processingunit 123 may give a weight thereto in consideration of the forgettingrate. Specifically, the statistical processing unit 123 adds work valuescalculated in the past after they are multiplied by a weightingcoefficient of 1 or less so that the influence of the work valuescalculated in the past may be small. In this manner, an accumulated workvalue is calculated. As the weighting coefficient, for example, anumerical value corresponding to a Gaussian distribution centered on thelatest point of time is used.

The statistical processing unit 123 may generate statistical data as atime-averaged value of the accumulated work value (hereinafter, referredto as an average work value) by dividing the accumulated work value bythe work period. Where weighting is performed in consideration of theforgetting rate, the statistical processing unit 123 generatesstatistical data by dividing the accumulated work value by an integralvalue of the weighting coefficient in the work period. The statisticalprocessing unit 123 may generate statistical data by setting arbitraryconditions, such as a specific worker, a specific date and time, aspecific season, a specific point of time, a specific area in the workarea or specific work content.

The statistical processing unit 123 may generate statistical data, basedon historical data. The historical data includes, for example,information on the postures of the worker calculated in the past, workvalues of the worker calculated in the past, statistical data generatedin the past, etc. Specifically, for example, data for the past week isaccumulated in the historical data. Thus, based on the data for the pastweek, the statistical processing unit 123 can generate statistical data,such as a cumulative value and an average work value of the work valuesof the past week, or a cumulative value and an average work value of thework values for each day of the week.

The display data generation unit 124 receives statistical data from thestatistical processing unit 123. The display data generation unit 124generates display data, based on the statistical data. Specifically, thedisplay data generation unit 124 uses display conversion data andconverts the statistical data into display data to be displayed in anexpression format that is easy for the user to recognize. Morespecifically, the display data generation unit 124 generates displaydata in which the accumulated work values included in the statisticaldata are displayed in correspondence to a plurality of body parts of ahuman body diagram regarded as a worker.

The display conversion data includes, for example, a human body diagramregarded as a worker, a GUI (Graphical User Interface) for displaying aplurality of human body diagrams in time series, etc.

The output device 400 is, for example, a monitor. The output device 400receives display data from the processing unit 120. The output device400 displays the display data. The output device 400 is not limited tothe monitor as long as the display data can be displayed. For example,the output device 400 may be a projector or a printer. The output device400 may include a speaker.

The work estimation apparatus 100 may include a memory and a processor(neither is shown). The memory stores, for example, various programsrelated to the operation of the work estimation apparatus 100 (e.g., awork estimation program that estimates a work of a worker). Theprocessor realizes each function of the acquisition unit 110, theprocessing unit 120 and the display control unit 130 by executingvarious programs stored in the memory.

The configuration of the work estimation system 1 and the workestimation apparatus 100 according to the first embodiment has beendescribed above. Next, the operation of the work estimation apparatus100 will be described with reference to the flowchart of FIG. 3.

FIG. 3 is a flowchart showing an operation example of the workestimation apparatus 100 according to the first embodiment. The processof the flowchart of FIG. 3 starts when the work estimation program isexecuted by the user.

(Step ST110)

When the work estimation program is executed, the acquisition unit 110acquires video data from the storage device 300. The acquisition unit110 outputs the acquired video data to the processing unit 120.

(Step ST120)

After the video data is acquired, the processing unit 120 calculates awork value related to the posture of the worker, based on the videodata. In addition, the processing unit 120 generates display data thatis based on the work value. In the description below, the processing ofstep ST120 is referred to as “work value calculation processing.” Aspecific example of the work value calculation processing will bedescribed with reference to the flowchart of FIG. 4.

FIG. 4 is a flowchart showing a specific example of the work valuecalculation processing according to the first embodiment. The flowchartof FIG. 4 illustrates details of the processing of step ST120 shown inFIG. 3.

(Step ST121)

After the video data is acquired, the posture estimation unit 121estimates a posture of the worker, based on the video data.Specifically, the posture estimation unit 121 detects a skeleton of aperson from video data, using a two-dimensional skeleton estimationmodel. The skeleton of the person can be represented, for example, bydata in which a plurality of key points in the person detected from thevideo data are associated with pixel coordinates (two-dimensionalcoordinates) of the video data. In the description below, the data inwhich a plurality of key points are associated with the two-dimensionalcoordinates will be referred to as a two-dimensional human skeletonmodel.

FIG. 5 is a diagram illustrating a two-dimensional human skeleton model11 according to the first embodiment. The two-dimensional human skeletonmodel 11 shown in FIG. 5 is represented, for example, by 18 key pointsKP1 to KP18 of the human body on the two-dimensional coordinates (P, Q).Key points KP1 to KP18 respectively correspond to “right eye”, “lefteye”, “right ear”, “left ear”, “nose”, “neck”, “right hand”, “lefthand”, “right elbow”, “left elbow”, “right shoulder”, “left shoulder”,“right hip”, “left hip”, “right knee”, “left knee”, “right foot” and“left foot.”

After the two-dimensional human skeleton model is generated, the postureestimation unit 121 generates a human skeleton model in three dimensions(three-dimensional human body skeleton model) by applying thetwo-dimensional human skeleton model to normalized three-dimensionalcoordinates, using a three-dimensional skeleton estimation model. Whenthe three-dimensional human skeleton model is generated from thetwo-dimensional human skeleton model, the posture estimation unit 121converts, for example, 18 key points of the two-dimensional humanskeleton model into 14 key points represented by three-dimensionalcoordinates. The 14 key points are respectively “head”, “neck”, “righthand”, “left hand”, “right elbow”, “left elbow”, “right shoulder”, “leftshoulder”, “right hip”, “left hip”, “right knee”, “left knee”, “rightfoot” and “left foot.” The key point “head” of the three-dimensionalhuman skeleton model may be estimated, for example, from five key pointsKP1 to KP5 of the two-dimensional human skeleton model, namely, “righteye”, “left eye”, “right ear”, and “left ear” and “nose”, or may beobtained by assuming the “nose” at key point KP5 as the “head.” In thedescription below, the key point “head” of the three-dimensional humanskeleton model will be represented as the key point KPH.

FIG. 6 is a diagram illustrating how a two-dimensional human skeletonmodel and a three-dimensional human skeleton model are generated from animage according to the first embodiment. The worker 13 shown in FIG. 6corresponds, for example, to a person detected from video data. Afterthe worker 13 is detected, the posture estimation unit 121 generates atwo-dimensional human skeleton model 15 corresponding to the worker 13by applying the two-dimensional skeleton estimation model 14 to thevideo data including the worker 13. At this time, the two-dimensionalhuman skeleton model 15 represented, for example, on the sametwo-dimensional coordinates (P, Q) as the video data.

Thereafter, the posture estimation unit 121 generates athree-dimensional human skeleton model 17 corresponding to the worker 13by applying the three-dimensional skeleton estimation model 16 to thetwo-dimensional human skeleton model 15. At this time, thethree-dimensional human skeleton model 17 is represented on thenormalized three-dimensional coordinates (Xn, Yn, Zn).

After the three-dimensional human skeleton model is generated, theposture estimation unit 121 estimates a behavior and a posture of theperson from the time series data on the three-dimensional human skeletonmodel, using a behavior estimation model. The estimation of the behaviorof the person is based on whether or not the person is moving (e.g.,“walking or moving”), and the estimation of the posture of the person isperformed irrespectively of whether or not the person is moving. Forexample, the posture is determined, based on how the states of aplurality of body parts are combined. Therefore, the states of theplurality of body parts have to be classified first. The states of thebody parts may be paraphrased as the postures of the body parts. In thedescription below, it is assumed that “estimation of posture” includes“estimation of human behavior.”

FIG. 7 is a table illustrating how states of a plurality of body partsare classified according to the first embodiment. In the table 19 shownin FIG. 7, body parts, state classification symbols and states are shownin association with each other. Specifically, the body parts are “back”,“upper limbs”, and “lower limbs.” The state classification symbolsdefine states of the body parts. The table 19 shown in FIG. 7 is basedon, but is not limited to, the OWAS (Ovako Working Posture AnalyzingSystem) method, which is one of the methods for evaluating the work loadof the worker. Although the influence of the weight or the force appliedto the worker can be taken into consideration in the OWAS method, theinfluence of the weight or the force applied to the worker will not betaken into consideration in the first and subsequent embodiments.

With respect to the “back”, four state classification symbols B1 to B4are listed. The four state classification symbols B1 to B4 respectivelycorresponds to “straight” (B1), “bent forward or backward” (B2),“twisted or bent sideways” (B3), and “twisted and bent sideways, or bentdiagonally forward” (B4).

With respect to the “upper limbs”, three state classification symbols U1to U3 are listed. The three state classification symbols U1 to U3respectively correspond to the state “both arms are below shoulder”(U1), the state “one arm is at shoulder height or above” (U2), and thestate “both arms are at shoulder height or above” (U3).

As for the “lower limbs”, seven state classification symbols L1 to L7are listed. The seven state classification symbols L1 to L7 respectivelycorrespond to the state “sitting” (L1), the state “standing with bothlegs straight” (L2), the state “standing with one leg with center ofgravity straight” (L3), the state “standing with both knees bent or insemi-crouching position” (L4), the state “standing with one leg withcenter of gravity bent or in semi-crouching position” (L5), the state“one or both knees are on floor” (L6), and the state “walking or moving”(L7).

Next, a detailed description will be given of a method in which thestates of a plurality of body parts are classified from athree-dimensional human skeleton model.

The states of the “back” of the three-dimensional human skeleton modelcan be classified, for example, by an angle by which the waist is bentand an angle by which the waist twisted. Specifically, the states of the“back” can be distinguished by detecting whether or not the waist isbent by 20 degrees or more and whether or not the waist is twisted by 20degrees or more.

The posture estimation unit 121 calculates an angle by which the waistis bent, based on angle θ1 formed by vector v1 and vector v2, the vectorv1 representing the direction from the midpoint of the hips (key pointKP13 of the “right hip” and key point KP14 of the “left hip”) of thethree-dimensional human skeleton model to the midpoint between the feet(key point KP17 of the “right foot” and key point KP18 of the “leftfoot”), and the vector v2 representing the direction from, the midpointof the hips to the neck (key point KP6 of the “neck”). The postureestimation unit 121 further calculates an angle by which the waist istwisted, based on angle θ2 formed by vector v3 and vector v4, the vectorv3 representing the direction from the right hip to the left hip of thethree-dimensional human skeleton model, and the vector v4 representingthe direction from the right shoulder (key point KP11 of the “rightshoulder”) to the left shoulder (key point KP12 of the “left shoulder”).The posture estimation unit 121 classifies (estimates) the states of the“back”, based on whether or not each of the angles θ1 and θ2 exceeds 20degrees.

The states of the “upper limbs” of the three-dimensional human skeletonmodel can be classified, for example, by the height of the right arm andthe height of the left arm. Specifically, the states of the “upperlimbs” can be distinguished by checking whether the right arm is abovethe shoulder height and whether the left arm is above the shoulderheight.

The posture estimation unit 121 detects whether the height-directioncoordinate of the right hand (key point KP7 of the “right hand”) orright elbow (key point KP9 of the “right elbow”) of thethree-dimensional human skeleton model is above the height-directioncoordinate of the right shoulder. The posture estimation unit 121further detects whether the height-direction coordinate of the left hand(key point KP8 of the “left hand”) or left elbow (key point KP10 of the“left elbow”) of the three-dimensional human skeleton model is above theheight-direction coordinate of the left shoulder. From these detections,the posture estimation unit 121 estimates the state of the “upperlimbs.”

The states of the “lower limbs” of the three-dimensional human skeletonmodel can be classified by detecting the position of the buttocks withrespect to the horizontal plane, the angle of the right leg and theangle of the left leg, the positions of the right foot and left footwith respect to the horizontal plane, the positions of the right kneeand left knee with respect to the horizontal plane and whether or notwalking is being performed. Specifically, the states of the “lowerlimbs” can be distinguished by detecting whether or not the buttocks areon the floor (or chair), whether or not the right leg is bent by 150degrees or less, whether or not the left leg is bent by 150 degrees orless, whether or not the right foot is in contact with the floor,whether or not the left foot is in contact with the floor, whether ornot the right knee is in contact with the floor, whether or not the leftknee is in contact with the floor, and whether or not the walkingmovement is being performed.

The posture estimation unit 121 calculates angle θ3 formed by vector v5and vector 6, the vector v5 representing the direction from the righthip to the right knee (key point KP15 of the “right knee”) of thethree-dimensional human skeleton model, and the vector v6 representingthe direction from the right knee to the right foot. Then, the postureestimation unit 121 determines whether or not the angle θ3 is 150degrees or less. The posture estimation unit 121 determines whether ornot the right foot is in contact with the floor by checking whether ornot the coordinates of the right foot of the three-dimensional humanskeleton model are above the floor (the height-direction coordinatevalue of which is, for example, zero). Similarly, the posture estimationunit 121 determines whether or not the right knee is in contact with thefloor by checking whether or not the coordinates of the right knee ofthe three-dimensional human skeleton model are above the floor. Theposture estimation unit 121 makes these determinations on the left sideof the body as well. Further, the posture estimation unit 121 determineswhether or not the worker is walking, based on the estimation of thebehavior of the worker. From these detections, the posture estimationunit 121 estimates the state of the “lower limbs.” In the presentembodiment, the state of sitting down is not assumed, so that thebuttocks are not in contact with the floor at all times.

As described above, posture estimation results using the behaviorestimation model are expressed as combinations of the states of aplurality of body parts. Specifically, a posture estimation resultcorresponds to a combination of state classification symbols of aplurality of body parts shown in the Table 19 of FIG. 7. In thedescription below, postures of workers estimated from images will bedescribed with reference to FIGS. 8 to 11. The workers in the imagesshown in FIGS. 8 to 11 are superimposed with a frame indicative ofworker detection, a posture estimation result, and a line and a brokenline indicative. of a three-dimensional human skeleton model.

FIG. 8 is a first example of how a posture of a worker is estimated froman image according to the first embodiment. FIG. 8 shows a worker whobends down to perform work. The posture estimation result 21 shown inFIG. 8 is “U1, L6, B4.” According to the posture estimation result 21,the posture of the worker shown in FIG. 8 is as follows: the back is inthe state of being “twisted and bent sideways, or bent diagonallyforward” (B4), the upper limbs are in the state where “both arms arebelow shoulder” (U1), and the lower limbs are in the state where “one orboth knees are on floor” (L6).

FIG. 9 is a second example of how a posture of a worker is estimatedfrom an image according to the first embodiment. FIG. 9 shows a workerwho bends down to perform work. The posture estimation result 23 shownin FIG. 9 is “U3, L6, B2.” According to the posture estimation result23, the posture of the worker shown in FIG. 9 is as follows: the back isin the state of being “bent forward or backward” (B2), the upper limbsare in the state where “both arms are at shoulder height or above”, andthe lower limbs are in the state where “one or both knees are on floor”(L6).

FIG. 10 is a third example of how a posture of a worker is estimatedfrom an image according to the first embodiment. FIG. 10 shows anoperator who works, with the center of gravity being on one leg. Theposture estimation result 25 shown in FIG. 10 is “U1, L5, B4.” Accordingto the posture estimation result 25, the posture of the worker shown inFIG. 10 is as follows: the back is in the state of being “twisted andbent sideways, or bent diagonally forward” (B4), the upper limbs are inthe state where “both arms are below shoulder” (U1), and the lower limbsare in the state “standing with one leg with center of gravity bent orin semi-crouching position” (L5).

FIG. 11 is a fourth example of how a posture of a worker is estimatedfrom an image according to the first embodiment. FIG. 11 shows a workerwho works with his upper body bent. The posture estimation result 27shown in FIG. 11 is “U1, L4, B4.” According to the posture estimationresult 27, the posture of the worker shown in FIG. 11 is as follows: theback is in the state of being “twisted and bent sideways, or bentdiagonally forward” (B4), the upper limbs are in the state where “botharms are below shoulder” (U1), and the lower limbs are in the state“standing with both knees bent or in semi-crouching position” (L4).

(Step ST122)

After the posture of the worker is estimated, the work value calculationunit 122 calculates a work value of the worker, based on the estimatedposture of the worker. Specifically, the work value calculation unit 122calculates a load value as the work value from the posture of theworker, by using a table in which a combination of states of a pluralityof body parts and load values are associated with each other.Alternatively, the work value calculation unit 122 may use a learnedmodel trained to calculate a work value from an estimated posture of theworker. The learned model mentioned here should preferably use a GNN,for example.

FIG. 12 is a table in which a combination of states of a plurality ofbody carts and load values are associated with each other in the firstembodiment. In Table 29, the column direction items show combinations ofthe state classification symbols of the “back” and state classificationsymbols of the “upper limbs”, the row direction items show stateclassification symbols of the “lower limbs, and load values are shown inthe load value group region 31 at the positions corresponding to theitems of the matrix. In the load value group region 31, four numbers(load values) 1 to 4 are assigned in ascending order of load value. Thetable 29 in FIG. 12 is based on the OWAS method, but this is notrestrictive.

Specifically, in the table 29, three state classification symbols U1 toU3 of the “upper limbs” are combined with four state classificationsymbols B1 to B4 of the “back.” Further, seven state classificationsymbols L1 to L7 of the “lower limbs” are associated with each of thesecombinations. That is, the number of items in the row direction is 12,the number of items in the column direction is 7, and the load valuegroup region 31 contains 84 load values.

For example, in the first example shown in FIG. 8, the postureestimation result 21 is “U1, L6, B4”, so that the combination of the“back” and the “upper limbs” is defined by “B4” and “U1”, and the “lowerlimbs” are in the state of “L6.” Thus, the load, value is “4” in theload value group region 31. Similarly, in the second example shown inFIG. 9, the posture estimation result 23 is “U3, L6, B2”, and the loadvalue is “4” in the load value group region 31. In the third exampleshown in FIG. 10, the posture estimation result 25 is “U1, L5, B4”, andthe load value is “4” in the load value group region 31. In the fourthexample shown in FIG. 11, the posture estimation result 27 is “U1, L4,B4”, and the load value is “4” in the load value group region 31.Therefore, the work value calculation unit 122 calculates load value “4”for each of the postures of the workers shown in FIGS. 8 to 11 by usingthe table 29.

Where the load values of a plurality of body parts are specified, theratio of load values relating to the plurality of body parts may beassociated with the load values in the load value group region 31. As analternative method, state classification symbols assigned to the statesof body parts shown in FIG. 7 and load values may be associated witheach other.

FIG. 13 is a diagram illustrating how types of load values are in thefirst embodiment. In the table 33 shown in FIG. 13, load states AC1 toAC4 are shown in correspondence to the load values 1 to 4 of the loadvalue group region 31. For example, AC1 means “The load which thisposture imposes on the musculoskeletal system is not a problem. Noimprovement is required.” AC2 means “This posture is harmful to themusculoskeletal system. Improvement is required early.” AC3 means “Thisposture is harmful to the musculoskeletal system. Improvement isrequired as soon as possible.” AC4 means “This posture is very harmfulto the musculoskeletal system. Improvement is required immediately.” Thetable 33 in FIG. 13 is based on the OWAS method, but this is notrestrictive.

(Step ST123)

After a work value of the worker is calculated, the statisticalprocessing unit 123 generates statistical data regarding the work valueof the worker, based on the work value. Specifically, the statisticalprocessing unit 123 generates, as statistical data, work valuesaccumulated from the start time of work to the current time or anaverage work value. Where a work value is a load value, the work valueis generated for each of a plurality of body parts (e.g., the back,upper limbs and lower limbs). In the statistical data, for example, theelapsed time from the start time of work is associated with the workvalue accumulated until the elapsed time or with the average work value.

(Step ST124)

After the statistical data is generated, the display data generationunit 124 generates display data, based on the generated statisticaldata. Specifically, the display data generation unit 124 generatesdisplay data in which accumulated work values included in thestatistical data are displayed in correspondence to a plurality of partsof a human body diagram regarded as a worker. In other words, thedisplay data generation unit 124 generates display data in which mapsrepresenting load values of a plurality of parts of the worker aresuperimposed on the human body diagram regarded as the worker.

FIG. 14 is a human body diagram 35 in which load values accumulated inthe first embodiment are displayed in correspondence to a plurality ofbody parts. The human body diagram 35 is divided into a back region 351,an upper limb region 352 and a lower limb region 353 as a plurality ofbody parts. These regions are color-coded according to the magnitude ofthe load value, and four load value levels LV1 to LV4 from the lowestload value to the highest load value are shown. Specifically, the backregion 351 is colored in the color corresponding to the load value levelLV4, the upper limb region 352 is colored in the color corresponding tothe load value level LV3, and the lower limb region 353 is colored inthe color corresponding to the load value level LV2. Thus, by looking atthe human body diagram 35, the user can recognize that the load of theback region 351 is higher than the loads of the other body parts of theworker.

In FIG. 14, the display data is a human body diagram in which theaccumulated load values at an arbitrary time are displayed incorrespondence to a plurality of body parts, but the display data is notlimited to this. For example, the display data may include a pluralityof human body diagrams in which the load values accumulated at aplurality of points of time are displayed in correspondence to aplurality of body parts.

FIG. 15 illustrates a plurality of human body diagrams 37, 39, 41 and 43in which load values accumulated in the first embodiment are displayedin correspondence to a plurality of body parts, the human body diagramsbeing arranged in time series. In FIG. 15, human body diagrams 37, 39,41 and 43 respectively corresponding to times t1 to t4 are shown.

In the human body diagram 37, the back region 371 is colored in thecolor corresponding to the load value level LV2, and the upper limbregion 372 and the upper limb region 373 are colored in the colorcorresponding to the load value level LV1. In the human body diagram 39,the back region 391 is colored in the color corresponding to the loadvalue level LV3, the upper limb region 392 is colored in the colorcorresponding to the load value level LV72, and the lower limb region393 is colored in the color corresponding to the load value level LV1.In the human body diagram 41, the back region 411 is colored in thecolor corresponding to the load value level LV4, the upper limb region412 is colored in the color corresponding to the load value level LV3,and the lower limb region 413 is colored in the color corresponding tothe load value level LV2. In the human body diagram 43, the back region431 and the upper limb region 432 are colored in the color correspondingto the load value level LV4, and the lower limb region 433 is colored inthe color corresponding to the load value level LV3. Thus, by looking atthe plurality of human body diagrams 37, 39, 41, 43 arranged in timeseries, the user can grasp how the loads of a plurality of body partsvary with the passage of work time.

The display data according to the first embodiment is not limited to theabove-mentioned data. For example, the display data may be created byprocessing the video data. At this time, the display data generationunit 124 superimposes maps representing load values of a plurality ofbody parts of the worker on the worker of the video data. Further,numerical values included in the statistical data, figures correspondingto the numerical values, and a table or a graph that is based on thestatistical data may be superimposed such that they are shown on theworker of the video data or in the neighborhood thereof.

(Step ST130)

After the display data is generated, the processing unit 120 outputs thedisplay data that is based on the work values to the output device 400.After the processing of step ST130, the work estimation program isended.

Where the video data is acquired in real time, the process flow mayreturn to step ST110 after the processing of step ST130, and thesubsequent processes may be repeated. The work estimation program may beended in response to an instruction by the user.

As described above, the work estimation apparatus 100 according to thefirst embodiment acquires video data relating to a predetermined area,calculates a work value of the work performed by the worker included inthe video data, based on the acquired video data, and displays thecalculated work value. This work value may include, for example, a loadvalue that represents the physical load which the work imposes or theworker. Further, the work estimation apparatus 100 according to thefirst embodiment may estimate a working posture of the worker, mayspecify a posture of a body part of the worker, and may superimpose amap representing a load value on a human body diagram regarded as aworker.

Therefore, the work estimation apparatus 100 according to the firstembodiment uses an image and enables the work state of a worker to bevisually grasped at a lower introduction cost than a conventional workvalue estimation using a sensor. In addition, the work estimationapparatus 100 according to the first embodiment can visualize the workload related to a body part of the worker, and work improvement of theworker is thus enabled.

Second Embodiment

In connection with the first embodiment, reference was made to the casewhere a work value (e.g., a load value) of the worker is calculated fromthe video data. On the other hand, in connection with the secondembodiment, a description will be given of the case where a work targetof the work performed by the worker is estimated.

FIG. 16 is a block diagram showing a configuration example of the workestimation system 1A including the work estimation apparatus 100Aaccording to the second embodiment. The work estimation system 1Aincludes a work estimation apparatus 100A, a photographing device 200A,a storage device 300A and an output device 400A. Since the photographingdevice 200A and the output device 400A are substantially similar to thephotographing device 200 and output device 400 of the first embodiment,a description thereof will be omitted.

The storage device 300A is a computer-readable storage medium thatstores data in a nonvolatile manner. The storage device 300A storesvideo data output from the photographing device 200A. Further, thestorage device 300A stores, for example, a plurality of data used in thework estimation apparatus 100A. The plurality of data of the secondembodiment include, for example, work target estimation data, inaddition to the plurality of data of the first embodiment. Details ofthe work target estimation data will be described later. The storagedevice 300A outputs video data and a plurality of data to the workestimation apparatus 100A in response to access from the work estimationapparatus 100A.

The work estimation apparatus 100A is, for example, a computer used by auser who manages the work estimation system 1A. The work estimationapparatus 100A includes an acquisition unit 110A, a processing unit 120Aand a display control unit 130A. The work estimation apparatus 100A mayinclude at least one of a photographing device 200A, a storage device300A and an output device 400A. Since the acquisition unit 110A and thedisplay control unit 130A are substantially similar to the acquisitionunit 110 and display control unit 130 of the first embodiment, adescription thereof will be omitted.

The processing unit 120A receives video data from the acquisition unit110A. The processing unit 120A calculates a work value related to aposture of the worker (working posture), based on the video data.Further, the processing unit 120A estimates a work target of the workperformed by the worker, based on the video data and the information onthe working posture of the worker. By accessing the storage device 300A,the processing unit 120A may receive a plurality of data necessary forprocessing the video data. The processing unit 120A may cause thestorage device 300A to store the calculated work value and the estimatedwork target information as they are, or may cause the storage device300A to store the calculated work value and the estimated work targetinformation in association with video data or video data information.

Further, the processing unit 120A generates display data in anexpression format that is easy for the user to recognize, based on thecalculated work value and the estimated work target. For example, thedisplay data of the second embodiment permits the statistical data ofthe calculated work value to be displayed for the work target shown in asketch of a work area. Specifically, the display data of the secondembodiment superimposes a map corresponding to statistical data on oneor more work targets in a two-dimensional or three-dimensional sketch.The processing unit 120A outputs the generated display data to thedisplay control unit 130A. The processing unit 120A may cause thestorage device 300A to store the generated display data as it is, or maycause the storage device 300A to store the generated display data inassociation with the video data or the video data information. Further,the display data may include numerical values included in thestatistical data, figures corresponding to the numerical values, and atable or a graph that is based on the statistical data, such that theyare shown on the sketch or in the neighborhood thereof.

FIG. 17 is a block diagram showing a configuration of the processingunit 120A of the work estimation apparatus 100A according to the secondembodiment. The processing unit 120A includes a posture estimation unit121A, a work value calculation unit 122A, a statistical processing unit123A, a display data generation unit 124A and a work target estimationunit 125. Since the work value calculation unit 122A is substantiallysimilar to the work value calculation unit 122 of the first embodiment,a description thereof will be omitted.

The posture estimation unit 121A estimates a posture cf a worker, basedon the video data. Specifically, the posture estimation unit 121Adetects a worker from the video data, using the posture estimation data,and estimates a posture of the detected worker. The posture estimationunit 121A outputs information on the estimated posture of the worker tothe work value calculation unit 122A and the work target estimation unit125.

The work target estimation unit 125 receives information on the postureof the worker from the posture estimation unit 121A. The work targetestimation unit 125 estimates a work target of the work performed by theworker, based on the video data, the posture of the worker and the worktarget estimation data. Specifically, the work target estimation unit125 identifies the position of the worker included in the video data ona sketch included in the work target estimation data, and estimates awork target of the work performed by the worker from a plurality of worktarget candidates associated with the sketch, based on the posture ofthe worker. The work target estimation unit 125 outputs information onthe estimated work target to the statistical processing unit 123A.

The work target estimation data includes, for example, a two-dimensionalsketch of a work area, a three-dimensional sketch of the work area, etc.The work target estimation data may include a region of coordinatesincluding the work target on the sketch, rectangular positioninformation on the work target, segmentation information on the worktarget, a name of the work target, etc.

The work target estimation unit 125 may detect, from the video data, therectangular position information on the work target, segmentationinformation thereon, or both. In this case, the work target estimationdata may include a learned learning model trained to detect an objectfrom the video data.

The statistical processing unit 123A receives a work value of the workerfrom the work value calculation unit 122A, and receives information onthe work target from the work target estimation unit 125. Thestatistical processing unit 123A generates statistical data regardingthe work value of the worker, based on the work value and the worktarget. Specifically, the statistical processing unit 123 generates, asstatistical data, work values accumulated from the start time of work toan arbitrary time, for each work target. The accumulated work values maybe generated, for example, for each of a plurality of body parts. Thestatistical processing unit 123A outputs the generated statistical datato the display data generation unit 124A. The statistical processingunit 123A may generate statistical data, based on historical data.

The display data generation unit 124A receives statistical data from thestatistical processing unit 123A, and receives information on the worktarget from the work target estimation unit 125. The display datageneration unit 124A generates display data, based on the statisticaldata and the information on the work target. Specifically, the displaydata generation unit 124A uses display conversion data and converts thestatistical data and the information on the work target into displaydata to be displayed in an expression format that is easy for the userto recognize. More specifically the display data generation unit 124Agenerates display data in which the accumulated work values included inthe statistical data are displayed in correspondence to the work targetshown in the sketch of the work area.

The display conversion data of the second embodiment includes, forexample, a two-dimensional sketch of the work area, a three-dimensionalsketch of the work area, and a GUI that displays a sketch and a humanbody diagram side by side, in addition to the display conversion data ofthe first embodiment.

The work estimation apparatus 100A may include a memory and a processor(neither is shown). The memory stores, for example, various programsrelated to the operation of the work estimation apparatus 100A (e.g., awork estimation program). The processor realizes each function of theacquisition unit 110A, the processing unit 120A and the display controlunit 130A by executing various programs stored in the memory. The workestimation program according to the second embodiment may include partor all of the processes of the work estimation program of the firstembodiment.

The configuration of the work estimation system 1A and the workestimation apparatus 100A according to the second embodiment has beendescribed above. Next, the operation of the work estimation apparatus100A will be described with reference to the flowchart of FIG. 18.

FIG. 18 is a flowchart showing an operation example of the workestimation apparatus 100A according to the second embodiment. Theprocess of the flowchart of FIG. 18 starts when the work estimationprogram is executed by the user.

(Step ST210)

When the work estimation program is executed, the acquisition unit 110Aacquires video data from the storage device 300A. The acquisition unit110A outputs the acquired video data to the processing unit 120A.

(ST220)

After the video data is acquired, the processing unit 120A calculates awork value related to the posture of the worker, based on the videodata, and estimates a work target of the work performed by the worker.In addition, the processing unit 120A generates display data that isbased on the work value and the work target. In the description below,the processing of step ST220 will be referred to as “work targetestimation processing.” A specific example of the work target estimationprocessing will be described with reference to the flowchart of FIG. 19.The work target estimation processing may include part or all of thework value calculation processing according to the first embodiment.

FIG. 19 is a flowchart showing a specific example of the work targetestimation processing according to the second embodiment. The flowchartof FIG. 19 illustrates details of the processing of step ST220 shown inFIG. 18. Since the processing of steps ST221 and ST222 is substantiallysimilar to the processing of steps ST121 and ST122 of the firstembodiment, a description thereof will be omitted.

(Step ST223)

After the work value of the worker is calculated, the work targetestimation unit 125 estimates a work target, based on the estimatedposture of the worker, the video data and the sketch. In the specificexample described below, the work target estimation unit 125 performsundermentioned processing, based on the video data captured by thephotographing device 200A arranged diagonally above the work area.First, an example of an image of video data and an example of atwo-dimensional sketch will be described with reference to FIGS. 20 and21.

FIG. 20 is a diagram illustrating how image 45 of video data accordingto the second embodiment. The image 45 shows a work area. The work areaincludes, for example, a working step 451, a pre-assembly product 452, aparts storage 453, a parts storage 454, an assembled product 455, aparts shelf 456 and a parts shelf 457. In the description below, each ofthe pre-assembly product 452, parts storage 453, parts storage 454,assembled product 455, parts shelf 456 and parts shelf 457 may beparaphrased as a work target.

The image 45 shows a worker 13 in the work area. The worker 13 is on theworking step 451 and faces the pre-assembly product 452. Thepre-assembly product 452 is placed on the working step 451. The workingstep 451, the parts storage 453, the parts storage 454, the assembledproduct 455, the parts shelf 456 and the parts shelf 457 are arranged inthe same plane (e.g., on the floor), with intervals therebetween.

FIG. 21 is a diagram illustrating a two-dimensional sketch 47 accordingto the second embodiment. The sketch 47 shows a work area. The sketch 47shows the working step 471, the pre-assembly product 472, the partsstorage 473, the parts storage 474, the assembled product 475, the partsshelf 476 and the parts shelf 477 such that the work area in the sketch47 corresponds to the work area shown in the image 45. Further, thepre-assembly product 472, the parts storage 473, the parts storage 474,the assembled product 475, the parts shelf 476 and the parts shelf 477correspond to one or more work target candidates. Where the position ofa work target candidate is specified from the sketch, a table may beused in which the work target candidate and the coordinate pointsindicative of the area of the work target candidate on the sketch areassociated with each other.

Next, the processing performed by the work target estimation unit 125will be described in detail. First, the work target estimation unit 125associates an image of video data with a two-dimensional sketch.Specifically, the work target estimation unit 125 acquires coordinatedata on reference points of the image 45 and coordinate data onreference points of the sketch 47, the reference points being commonportions in the work area. Those coordinate data may be included in thework target estimation data in advance.

FIG. 22 is a diagram illustrating three reference points in the image 45shown in FIG. 20. In 22, the image 45 is arranged on two-dimensionalcoordinates (Q, P). Three reference points S1 to S3 are shown on theimage 45. These three reference points S1 to S3 are determined such thatthe floor in the work area can be defined. Specifically, the threereference points S1 to S3 are determined such that they do not line upin a plane corresponding to the floor of the work area.

FIG. 23 is a diagram illustrating the three reference points in thesketch shown in FIG. 21. In FIG. 23, the sketch 47 is arranged ontwo-dimensional coordinates (I, J). Similar to the image 45 of FIG. 22,three reference points S1 to S3 are shown on the sketch 47. Thereference points in FIG. 23 are determined such that they correspond tothe reference points in FIG. 22.

FIG. 24 is a diagram for illustrating how the image 45 shown in FIG. 20and the sketch 47 shown in FIG. 21 are related to virtualthree-dimensional space 49. The work target estimation unit 125 usesperspective projection transformation such that the three referencepoints in the image 45 and the three reference points in the sketch 47coincide with each other on the virtual three-dimensional space 49.Specifically, the work target estimation unit 125 projects the sketch 47in parallel in a zero-height XY plane of the virtual three-dimensionalspace 49. Then, the work target estimation unit 125 projects the virtualthree-dimensional space 49 onto a two-dimensional plane by use of avirtual camera, and arranges the virtual three-dimensional space 49 suchthat the parallel-projected reference points on the sketch coincide withthe positions of the reference points in the image 45.

FIG. 25 is a diagram illustrating how the coordinates of the referencepoints in the image 45 and the coordinates of the reference points inthe sketch 47 are converted to coordinates of reference points in thevirtual three-dimensional space 49 according to the second embodiment.In the table 51 shown in FIG. 25, the coordinates of the three referencepoints S1 to S3 in the image 45 and those in the sketch 47 areassociated with each other. The coordinates of the reference points S1to S3 of the image 45 are indicated by (P1, Q1), (P2, Q2) and (P3, Q3),respectively, and the coordinates of the reference points S1 to S3 ofthe sketch. 47 are indicated by (I1, J1), (I2, J2) and (I3, J3),respectively.

The work target estimation unit 125 converts the coordinates of thereference points shown in the table 51 into the coordinates in thevirtual three-dimensional space 49; based on the perspective projectiontransformation 53. The table 55 in FIG. 25 shows how the coordinates ofthe three reference points S1 to S3 are in the virtual three-dimensionalspace 49. The coordinates of the three reference points S1 to S3 in thevirtual three-dimensional space 49 are indicated by (X1, Y1, Z1) (X2,Y2, Z2) and (X3, Y3, Z3), respectively. Since these three referencepoints S1 to S3 are determined in the zero-height XY plane of thevirtual three-dimensional space 49, the values of Z1, Z2 and Z3indicative of the height are all zero.

The working step 471 is not at the same height as the floor in the workarea. Therefore, a reference point that can define the working step 471is determined in both the image and the sketch, and is associateddifferently from the reference points on the floor in the work area. Inthe description below, it is assumed that the floor in the work area andthe working step 471 are distinguished and the image of the video dataand the two-dimensional sketch are associated with each other.

After the image of the video data and the two-dimensional sketch areassociated with each other, the work target estimation unit 125 arrangesa three-dimensional human skeleton model represented by normalizedthree-dimensional coordinates in the virtual three-dimensional space.Specifically, the work target estimation unit 125 converts thenormalized three-dimensional coordinates of the three-dimensional humanskeleton model used for the posture estimation into coordinates of thevirtual three-dimensional space.

FIG. 26 is a diagram for illustrating how the three-dimensional humanskeleton model 17 represented by the normalized three-dimensionalcoordinates is arranged in the virtual three-dimensional space 49according to the second embodiment. The work target estimation unit 125converts the three-dimensional human skeleton model 17 into athree-dimensional human skeleton model 59 represented in the virtualthree-dimensional space 49, based on coordinate conversion 57. Thecoordinate conversion 57 utilizes the association between the image 45and the virtual three-dimensional space 49.

Specifically, the work target estimation unit 125 uses the key pointsKP17 and KP18 of the three-dimensional human skeleton model 17 and thecorresponding coordinates on the image 45, and identifies thecoordinates of the key points KP17 and KP18 in the virtualthree-dimensional space 49. Since the key points KP17 and KP18correspond to the “right foot” and the “left foot”, respectively, theycan be reference points of the three-dimensional human skeleton model17. Thereafter, the work target estimation unit 125 calculatescoordinates of each key point of the three-dimensional human skeletonmodel 59 from the coordinates of each key point of the three-dimensionalhuman skeleton model 17, based on the identified coordinates of the keypoints KP17 and KP18 in the specified virtual three-dimensional space49.

FIG. 27 is a diagram illustrating how the coordinates of key points ofthe three-dimensional human skeleton model 17 represented by thenormalized three-dimensional coordinates are converted into thecoordinates of the key points of the three-dimensional human skeletonmodel 59 represented in the virtual three-dimensional space according tothe second embodiment. In the table 61 shown in FIG. 27, the coordinatesof the 14 key points KPH, KP6, KP7, . . . , KP18 of thethree-dimensional human skeleton model 17 are shown. The coordinates ofthe key points KPH, KP6, KP7, . . . , KP18 in the normalizedthree-dimensional coordinates are (Xn1, Yn1, Zn1), (Xn6, Yn6, Zn6),(Xn7,Yn7, Zn7), . . . , (Xn18, Yn18, Zn18), respectively. In the table65 shown in FIG. 27, the coordinates of the 14 key points KPH, KP6, KP7,. . . , KP18 of the three-dimensional human skeleton model 59 are shown.The coordinates of the key points KPH, KP6, KP7, . . . , KP18 in thevirtual three-dimensional space 49 are (X10, Y10, Z10), (X60, Y60, Z60),(X70, Y70, Z70), . . . , (X180, Y180, Z180), respectively.

The work target estimation unit 125 converts the coordinates shown inthe table 61 into the coordinates shown in the table 65, based oncoordinate conversion 63. The coordinate conversion 63 is similar to thecoordinate conversion 57.

After the three-dimensional human skeleton model represented by thenormalized three-dimensional coordinates is arranged in the virtualthree-dimensional space, the work target estimation unit 125 estimatesthe direction of the worker represented by the three-dimensional humanskeleton model. Specifically, the work target estimation unit 125calculates a direction of the vector corresponding to the direction ofthe worker from the coordinates of the key points of thethree-dimensional human skeleton model arranged in the virtualthree-dimensional space.

FIG. 28 is a diagram for illustrating how the direction of the worker isestimated according to the second embodiment. FIG. 28 shows how a workerrepresented by the three-dimensional human skeleton model 59 looks likewhen viewed in the height direction (Z-axis direction) of the virtualthree-dimensional space. The work target estimation unit 125 calculatesa direction of the vector v7 representing the front direction 67 of theworker from the coordinates of the key points KP11 and KP12 of thethree-dimensional human skeleton model 59. Specifically, the work targetestimation unit 125 uses the coordinates (X110, Y110, Z110) of the keypoint KP11 and the coordinates (X120, Y120, Z120) of the key point KP12in the virtual three-dimensional space and calculates a direction of thevector v7 from the midpoint between the coordinates (X110, Y110) and thecoordinates (X120, Y120) in the XY plane. The direction of the vector v7can be represented by ((−1/(Y110−Y120)), 1/(X110−X120)).

Where the direction of the worker is estimated on a three-dimensionalsketch, the work target estimation unit 125 calculates a directionvector representing the front direction of the body of the worker bycalculating a vector v10 represented by the outer product of a vector v8representing the direction from the midpoint between the hips to theright shoulder in the three-dimensional human skeleton model 59 and adirection vector v9 from that midpoint to the left shoulder.

After the direction of the worker represented by the three-dimensionalhuman skeleton model is estimated, the work target estimation unit 125estimates a work target of the work performed by the worker from one ormore work target candidates, based on the direction of the worker andthe sketch in which one or more work target candidates are associated.

FIG. 29 is a diagram for illustrating how the direction of the worker isassociated with the work targets in the sketch according to the secondembodiment. FIG. 29 shows how the sketch 69 projected on the virtualthree-dimensional space looks like when viewed in the height direction.FIG. 29 shows the sketch 69 and a worker 71 on the sketch 69. In thesketch 69, a work target candidate point 731 indicative of a work targetis associated as a coordinate point at an arbitrary position in theregion of the pre-assembly product 73. The work target candidate point731 is associated with the pre-assembly product 73. It is assumed thatthe position and direction of the worker 71 are already specified in thevirtual three-dimensional space.

FIG. 29 shows a fan-shaped work target area 711 in which the position ofthe worker 71 is an apex and the direction of the worker 71 (e.g., thedirection of the vector v7 in FIG. 28) is a reference. The fan-shapedarc shown as the work target area 711 is determined based on, forexample, the physical characteristics of the worker. The physicalcharacteristics of the worker are, for example, the arm length of aspecific worker or the average arm length of a plurality of workers.Preferably, the work target area 711 is an area within the reach of bothhands of the operator. The work target estimation unit 125 estimatesthat the pre-assembly product 73 associated with the work targetcandidate point 731 is a work target by specifying the work targetcandidate point 731 included in the work target area 711. The worktarget area is not limited to the fan shape, and may be a circle, arectangle, or the like.

Although not depicted in the example shown in FIG. 29, work targetcandidate. points are associated with other work target candidates(e.g., a parts storage and a parts shelf) as coordinate points atarbitrary positions in the. areas indicative of the work targetcandidates. As the work target candidate points, the positions that canbe associated with the work target candidates may be determined inconsideration of the range in which the work target areas overlap.

The estimation of the work target is not limited to the example shown inFIG. 29. For example, the work target estimation unit 125 may use partor all of the area indicative of a work target candidate as a worktarget candidate area, and may estimate a work target according to thedegree of agreement with the work target area of the worker.

If the direction of the worker is associated with the work target shownin the sketch on a three-dimensional sketch, the work target estimationunit 125 determines a triangle defined by three points in thethree-dimensional human skeleton model (worker 71), which are thecoordinate point of the right shoulder, the coordinate point of theright shoulder and the coordinate point of the midpoint of the hips, anddetermines an axis extending from the center of gravity of that trianglein the direction of the vector v10. Then, the work target estimationunit 125 calculates a conical three-dimensional region, a sphericalthree-dimensional region or a rectangular three-dimensional regioncentered on the determined axis, as a work target area. In analternative example, the work target estimation unit 125 may calculatespherical three-dimensional regions centered on the right hand and theleft hand of the three-dimensional human skeleton model, as work targetareas.

(Step ST224)

After the work value of the worker is calculated and the work target isestimated, the statistical processing unit 123A generates statisticaldata regarding the work value of the worker, based on the calculatedwork value of the worker and the estimated work target specifically, thestatistical processing unit 123A generates, as statistical data, workvalues accumulated from the start time of work to an arbitrary point oftime or an average work value, for each work target. Alternatively, thestatistical processing unit 123A may generate statistical data in whicha load value level and a work time are associated with each other foreach work target. In the specific example described below, it is assumedthat the statistical data is data in which a load value level and a worktime are associated with each other for each work target.

(Step ST225)

After the statistical data is generated, the display data generationunit 124A generates display data, based on the generated statisticaldata. Specifically, the display data generation unit 124A generatesdisplay data in which a load value map corresponding to the generatedstatistical data is superimposed on the sketch.

The load value map is represented, for example, by a combination ofcircles which are based on respective work targets. The radial length ofeach circle corresponds to the work time according to the load valuelevel, and the shade of each circle corresponds to a load value level.The load value map may be paraphrased as a color map or a heat map.

FIG. 30 is a sketch 75 illustrating how accumulated load values aredisplayed in correspondence to a plurality of work targets according tothe second embodiment. The load value map 751 is superimposed on thesketch 75. The load value map 751 includes, for example, four load areas752 to 755 corresponding to the load value levels. The load area 752corresponds to the load value level LV1 and has the widest contour. Theload area 753 corresponds to the load value level LV2 and has a contournarrower than the load areas 752. The load area 754 corresponds to theload value level LV3 and has a contour narrower than the load area 753.The load area 755 corresponds to the load value level LV4 and has acontour narrower than the load area 754. The contour of the load valuemap 751 is the same as the contour of the load area 752.

Specifically, in the sketch 75 shown in FIG. 30, the load area 755 areshown with respect to the parts storage located at the upper right(corresponding to the parts storage 473 in the sketch 47 shown in FIG.21), the pre-assembly product, and the parts shelf located at the lowerleft (corresponding to parts shelf 477 in the sketch 47 shown in FIG.21). Therefore, the user who looks at the sketch 75 can recognize thatthe work performed for these work targets imposes a heavy load on theworker. The sketch 75 on which the load value map 751 is superimposedmay be paraphrased as display data.

In FIG. 30, the sketch 75 in which the accumulated load values aredisplayed correspondence to a plurality of work targets is used as thedisplay data, but the display data not limited to this. For example, thedisplay data may be data that includes a human body diagram showing aspecific body part and a sketch showing a load value related to thespecific body part in correspondence to a plurality of work targets.

FIG. 31 is a diagram illustrating a first specific example of displaydata 77 obtained according to the second embodiment and including ahuman body diagram 812 showing a specific body part and a sketch 79showing load values related to the specific body part and displayed incorrespondence to a plurality of work targets. The display data 77 is aGUI including the sketch 79, a pull-down menu 811 and the human bodydiagram 812.

When the user selects, for example, the body part “back” from thepull-down menu 811, the back region is colored in the human body diagram812, and the load value map 791 regarding the “back” is superimposed onthe sketch 79. At this time, the load area 792 corresponding to the loadvalue level LV4 is displayed in the parts storage (corresponding to theparts storage 473 in the sketch 47 shown in FIG. 21) located at theupper right. Therefore, the user who looks at the sketch 77 canrecognize that the work performed for the parts storage imposes a heavyload on the “back” of the worker.

FIG. 32 is a diagram illustrating a second specific example of displaydata 83 obtained according to the second embodiment and including ahuman body diagram 872 showing a specific body part and a sketch 85showing load values related to the specific body part and displayed incorrespondence to a plurality of work targets. The display data 83 is aGUI including the sketch 85, a pull-down menu 871 and the human bodydiagram 872.

When the user selects, for example, the body part “upper limbs” from thepull-down menu 871, the upper limb regions are colored, in the humanbody diagram 872, and the load value map 851 regarding the “upper limbs”is superimposed on the sketch 85. At this time, the load area 852corresponding to the load value level LV4 is displayed on thepre-assembly product. Therefore, the user who looks at the sketch 83 canrecognize that the work performed for the pre-assembly product imposes aheavy load on the “upper limbs” of the worker.

FIG. 33 is a diagram illustrating a third specific example of displaydata 89 obtained according to the second embodiment and including ahuman body diagram 932 showing a specific body part and a sketch 91showing load values related to the specific body part and displayed incorrespondence to a plurality of work targets. The display data 89 is aGUI including the sketch 91, a pull-down menu 931 and the human bodydiagram 932.

When the user selects, for example, the body part “lower limbs” from thepull-down menu 931, the lower limb regions are colored in the human bodydiagram 932, and the load value map 911 regarding the “lower limbs” issuperimposed on the sketch 91. At this time, the load area 912corresponding to the load value level LV4 is displayed in the partsshelf (corresponding to the parts shelf 477 in the sketch 47 shown inFIG. 21) located at the lower left. Therefore, the user who looks at thedisplay data 89 can recognize that the work performed for the partsshelf located at the lower left imposes a heavy load on the “lowerlimbs” of the worker.

In FIGS. 31 to 33, the display data is data including a human bodydiagram showing a specific body part and a sketch showing load valuesrelated to the specific body part in correspondence to a plurality ofwork targets, but the display data is not limited to this. For example,the display data may be data including a sketch diagram showing aspecific work target and a human body diagram in which the accumulatedload values related to the specific work target are displayed incorrespondence to a plurality of body parts of the worker.

FIG. 34 is a diagram illustrating display data 95 including a sketch 97showing a specific work target and a human body diagram 99 in whichaccumulated load values are displayed in correspondence to a pluralityof body parts. The display data 95 is a GUI including the sketch 97 andthe human body diagram 99. For example, when the user selects the partsstorage 971 located at the upper right from the sketch 97, the region ofthe parts storage 971 is colored in the sketch 97, and in the human bodydiagram 99, the body part regions are color-coded according to theaccumulated load values corresponding to them. Therefore, the user wholooks at the display data 95 can confirm the load values correspondingto the body parts of the worker in relation to a specific work target.

The display data according to the second embodiment is not limited tothe above-mentioned data. For example, the display data may berepresented by a three-dimensional sketch. At this time, the displaydata generation unit 124A generates display data in which a load valuemap is superimposed on the three-dimensional sketch. In this case, theload value map is superimposed, for example, on the surface of thethree-dimensional model of the work targets.

In addition, the display data may be generated by processing the videodata. At this time, the display data generation unit 124A superimposes aload value map on the work targets on the video data. Further, numericalvalues included in the statistical data, figures corresponding to thenumerical values, and a table or a graph that is based on thestatistical data, may be superimposed such that they are shown on thework targets of the video data or in the neighborhood thereof.

(Step ST230)

After the display data is generated, the processing unit 120A outputsthe display data that is based on the work values and work target to theoutput device 400A. After the processing of step ST230, the workestimation program is ended.

Where the video data is acquired in real time, the process flow mayreturn to step ST210 after the processing of step ST230, and thesubsequent processes may be repeated. The work estimation program may beended in response to an instruction by the user.

As described above, the work estimation apparatus 100A according to thesecond embodiment acquires video data relating to a predetermined area,calculates a work value of the work performed by the worker included inthe video data, based on the acquired video data, and displays the workvalue. This work value may include, for example, a load value thatrepresents the physical load which the work imposes on the worker.Further, the work estimation apparatus 100A according to the secondembodiment may estimate a working posture of the worker, may specify aposture of a body part of the worker, and may superimpose a maprepresenting a load value on a human body diagram regarded as a worker.Still further, the work estimation apparatus 100A according to thesecond embodiment may estimate a work target of the work performed bythe worker, and superimpose a map showing the load value on the sketchshowing the estimated work target.

Therefore, the work estimation apparatus 100A according to the secondembodiment uses an image and enables the work state of a worker to bevisually recognized at a lower introduction cost than a conventionalwork value estimation using a sensor. In addition, the work estimationapparatus 100A according to the second embodiment can visualize the workload related to a body part of the worker in relation to the worktarget, and can help improve the working environment from the viewpointof safety and health.

FIG. 35 is a block diagram illustrating a hardware configuration of acomputer 500 according to one embodiment. The computer 500 includes aCPU (Central Processing Unit) 510, a RAM (Random Access Memory) 520, aprogram memory 530, an auxiliary storage device 540 and an input/outputinterface (input/output I/F) 550. These elements are provided ashardware. The CPU 510 communicates with the RAM 520, the program memory530, the auxiliary storage device 540 and the input/output interface 550via a bus.

The CPU 510 is an example of a general-purpose processor. The RAM 520 isused as a working memory by the CPU 510. The RAM 520 includes a volatilememory such as an SDRAM (Synchronous Dynamic Random Access Memory). Theprogram memory 530 stores various programs including a signal processingprogram. As the program memory 530, for example, a ROM (Read OnlyMemory), a portion of the auxiliary storage device 540, or a combinationof these is used. The auxiliary storage device 540 stores data in anonvolatile manner. The auxiliary storage device 540 includes anonvolatile memory such as an HDD or an SSD.

The input/output interface 550 is an interface for coupling to anotherdevice. The input/output interface 550 is used, for example, forcoupling to the photographing device, storage device and output deviceshown in FIGS. 1 and 16.

Each of the programs stored in the program memory 530 includes computerexecutable instructions. When the program (computer executableinstruction) is executed by the CPU 510, it causes the CPU 510 toexecute a predetermined process. For example, when the work estimationprogram is executed by the CPU 510, the CPU 510 executes a series ofprocesses described in relation to the acquisition unit, the processingunit and the display control unit.

The program may be provided to the computer 500 in a state of beingstored in a computer-readable storage medium. In this case, for example,the computer 500 further includes a drive (not shown) that reads datafrom the storage medium, and acquires the program from the storagemedium. Examples of storage media include a magnetic disk, optical disks(CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.)and a semiconductor memory. The program may be stored in a server on acommunication network such that the computer 500 can download theprogram from the server using the input/output interface 550.

The processes described in connection with the embodiments are notlimited to those which a general-purpose hardware processor such as aCPU 510 executes according to a program, and may be performed by adedicated hardware processor such as an ASIC (Application SpecificIntegrated Circuit). The term processing circuit (processing unit)includes at least one general-purpose hardware processor, at least onededicated hardware processor, or a combination of at least onegeneral-purpose hardware processor and at least one dedicated hardwareprocessor. In the example shown in FIG. 35, the CPU 510, the RAM 520 andthe program memory 530 correspond to the processing circuit.

Therefore, according to each of the above embodiments, the work of aworker can be estimated without imposing a burden on the worker.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. A work estimation apparatus comprising processingcircuitry configured to: acquire video data on a predetermined area;calculate a work value related to work performed by a worker included inthe video data, based on the video data; and display the work value. 2.The apparatus according to claim 1, wherein the work value includes aload value that represents a physical load which the work imposes on theworker.
 3. The apparatus according to claim 2, wherein the processingcircuitry is further configured to: calculate a posture feature amountof the worker by applying a skeleton estimation model to the video data;estimate a working posture of the worker, based on the posture featureamount; and calculate the load value, based on the working posture. 4.The apparatus according to claim 3, wherein the processing circuitry isfurther configured to: specify a plurality of body parts of the workerfrom the posture feature amount; and estimate the working posture byspecifying a posture of each of the plurality of body parts by using atable in which the plurality of body parts and a plurality of posturescorresponding to the plurality of body parts are associated with eachother.
 5. The apparatus according to claim 3, wherein the processingcircuitry is further configured to calculate the load value by use of atable in which a plurality of working postures and a plurality of loadvalues are associated with each other.
 6. The apparatus according toclaim 3, wherein the work value includes a plurality of load valuescorresponding to a plurality of body parts of the worker in relation tothe work, and the processing circuitry is further configured to:identify the plurality of body parts from the posture feature amount;identify the posture of each of the plurality of body parts by use of atable in which the plurality of body parts and a plurality of posturescorresponding to the plurality of body parts are associated with eachother; and calculate the plurality of load values, based on the posturesof the plurality of body parts.
 7. The apparatus according to claim 6,wherein the processing circuitry is further configured to: superimpose amap representing load values of the plurality of body parts on a humanbody diagram regarded as the worker; and display the human body diagramon which the map is superimposed.
 8. The apparatus according to claim 3,wherein the work value includes a plurality of load values expressed intime series, the processing circuitry is further configured to:calculate the plurality of load values expressed in time series; andgenerate statistical data on load values of the worker, based on theplurality of load values expressed in time series.
 9. The apparatusaccording to claim 3, wherein the processing circuitry is furtherconfigured to estimate a work target of the work performed by the workerfrom among one or more work target candidates, based on a sketch of thepredetermined area in which the working posture and one or more worktarget candidates are associated with each other.
 10. The apparatusaccording to claim 9, wherein the processing circuitry is furtherconfigured to present the work target and the load value in associationwith each other.
 11. The apparatus according to claim 10, wherein theprocessing circuitry is further configured to: superimpose a maprepresenting the load values on the work target on the sketch; anddisplay the sketch on which the map is superimposed.
 12. The apparatusaccording to claim 9, wherein the processing circuitry is furtherconfigured to present the work target and the load values correspondingto a plurality of body parts of the worker in association with eachother.
 13. The apparatus according to claim 12, wherein the processingcircuitry is further configured to: superimpose a map representing loadvalues of a plurality of body parts of the worker on the work target onthe sketch; and display the sketch on which the map is superimposed. 14.The apparatus according to claim 9, wherein the work value includes aplurality of load values expressed in time series, and the processingcircuitry is further configured to: calculate the plurality of loadvalues expressed in time series; estimate a plurality of work targets,based on the video data, the sketch, and the working posture; andgenerate statistical data regarding the load value of the worker foreach work target of the plurality of work targets, based on theplurality of load values expressed in time series and the plurality ofwork targets.
 15. The apparatus according to claim 9, wherein the workvalue includes a plurality of load values expressed in time series, thesketch is associated with a plurality of work target items in advance,and the processing circuitry is further configured to: calculate theplurality of load values expressed in time series, based on the videodata; and generate statistical data regarding the load value of theworker for each of the plurality of work target items.
 16. The apparatusaccording to claim 1, wherein the processing circuitry is furtherconfigured to cause a storage device to store the work value and videodata information on the video data in association with each other. 17.The apparatus according to claim 9, wherein the processing circuitry isfurther. configured to cause a storage device to store the work value,the work target, and video data information on the video data inassociation with each other.
 18. The apparatus according to claim 8,wherein the processing circuitry is further configured to generate, asthe statistical data, one of sums of the load values expressed in timeseries; an average value of the sums, a weighted average value of thesums, weighted sums of the plurality of load values expressed in timeseries, an average value of the weighted sums, and a weighted averagevalue of the weighted sums.
 19. A work estimation method comprising:acquiring video data on a predetermined area, calculating a work valuerelated to work performed by a worker included in the video data, basedon the video data, and displaying the work value.
 20. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to execute processing comprising: acquiring video data on apredetermined, area; calculating a work value related to work performedby a worker included in the video data, based on the video data,displaying the work value.